Top quark rare decays in the standard model and beyond 
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Abstract 

Theoretical predictions for the top quark rare decays are reviewed within and beyond the standard 
model. Expectations at the CERN Large Hadron Collider are discussed. 

1 Introduction 

The production of 10^ — 10^ top quark pairs per year at the CERN Large Hadron Collider (LHC) will 
' allow to probe the top couplings to both known and new particles involved in possible top decay channels 

, different from the main t bW. Thanks to the large top mass, there are several decays that can be 

considered, even involving the presence of on-shell heavy vector bosons or heavy new particles in the final 
, states. On a purely statistical basis, one should be able to detect a particular decay channel whenever its 

^ ' branching ratio (B) is larger than about 10~® — 10~^. In practice, background problems and systematics 

will lower this potential by a few orders of magnitude, the precise reduction being dependent of course 
' from the particular signature considered. We will see, that the final detection threshold for each channel 

. will not allow the study of many possible final states predicted in the standard model (SM), unless new 

stronger couplings come into play. 

After discussing in section 2 the main top quark rare decay channels predicted by the SM, we review 
in section 3 the expectations of a large class of possible extensions of the SM, such as the inclusion of 
, a 4"* fermion family, the two-Higgs-doublet models, and the minimal supersymmetric standard model 

• (MSSM). Finally, results of the analysis of the LHC potential in the top rare decays field are reported in 

, section 4. 

o . 
o ■ 

O ; 2 Standard Model Decays 

^: . . . . 

CIh! In this section, we give an overview of the decay channels of the top quark in the framework of the SM. 

^ • In the SM the decay t bW is by far the dominant one. The top total width is then given by: 

(D : ^3 

-p.' Tt ^r{t^bW):^ 0.17 GeV\Vtbf^^l.55GeY. (1) 

^ ! w 

k> , The rates for other decay channels are predicted to be smaller by a few orders of magnitude in the SM. The 

■ second most-likely decays are the Cabibbo-Kobajashi-Maskawa (CKM) non-diagonal decays t — > sM^and 

5^ I t ^ dW. Assuming \Vts\ — 0.04 and \Vtd\ — 0.01, respectively [0, one gets; 

B{t sW) - 1.6 X 10"^ and B{t dW) - 1 x lO"'^, (2) 

in the SM with three families. From now on, B{t —>...) for a generic decay channel, stands for the 
quantity: 

The two-body tree-level decay channels are the only ones that LHC could be able to detect in the 
framework of the SM. The next less-rare processes indeed turn out to have rates not larger than 10"^. 
At tree level, the decay t bWZ (fig. |l]) has some peculiar features, since the process occurs near the 
kinematical threshold (mj ~ Mw + Mz + rrn,). This fact makes crucial the W and Z finite- width effects 
in the theoretical prediction of the corresponding width. In |^ (see also [^), an estimate of B{t bWZ) 
was given on the basis of the definition: 

T{t ^ bWZ) ^ , r(^-^M^.ee) ^ 

^ ' B{W /ii^p)B(Z ^ ee) ^ ^ 
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Figure 1: Feynman graphs for the decay t bWZ {t — > bWH). 



involving experimentally well-observable decays. This definition involves 14 diagrams contributing to the 
hjjLVfiee final state in the SM. The estimate for the corresponding branching ratio is: 

B,^t{t^bWZ):^&xlQ-\ (5) 

for mt = 175 GeV, assuming a minimum cut of O.SA/^ on the ee-pair invariant mass. This cut tries to 
cope with the contribution of background graphs where the ee pair rises not from a Z boson but from 
a photon. Recently, in [Q, it has been argued that B{t bWZ) should be defined by including in the 
definition in eq. only the three (gauge-invariant) diagrams that correspond to the process where the 
final state bfiiy^ee is truly mediated by a and a Z. Then one obtains, for mt = 175 GeV: 

B{t^bWZ) = Bres{t ^bWZ) = 2x10-^. (6) 

Such an increase in B{t ^ bWZ) is partly due to the negative interference effects with the graphs where 
a photon replaces the Z boson (which are present in the previous estimate of the quantity), and partly 
to the absence of any kinematical cut (which for the resonant graphs is not needed). 
One could also find less ambiguous definitions for B{t bWZ) than the one involved in eq. (^. For 
instance, the new definition: 

Tit ^ bwz) ^ r(t^bM. ....) — ^ 

^ ^ B{W ^lv^)B{Z ^ v,v,) ^ ' 

would not require any kinematical cut, would involve a negligible background, and would give for B{t — > 
bWZ) the same result as in eq.(||). Of course, the signature b^v^^VgVe would not be practical from an 
experimental point of view, also because of the tiny rates involved in this decay channel. 
Then, the authors in agree that an experimental measurement of the decay rate oit^ bWZ should 
go through the definition in eq. (^, where the final state bjiv^ee includes all the possible backgrounds. 
They confirm the result in eq. (^) as an experimental effective quantity. 

If the Higgs boson is light enough, one could also have the decay t bWH (fig. although the 
present limits on mn strongly suppress its rate. For mn ^ 100 GeV, one gets (see also [||): 

B{t bWH) < 7 X 10-**. (8) 

Finally, the decay t cWW is very much suppressed by a GIM factor -t-t^- in the amplitude. One than 
gets g: 

B{t cWW) ~ IQ-i^ (9) 
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Figure 2: Feynman graphs for the decay t cH in the unitary gauge (rric = is assumed). 



One can also consider the radiative three-body decays t bWg and t bWj. These channels suffer 
from infrared divergences, and the evaluation of their rate requires a full detector simulation, including 
for instance the effects of the detector resolution and the jet isolation algorithm. In an idealized situation 
where the rate is computed in the t rest frame with a minimum cut of 10 GeV on the gluon or photon 
energies, one finds |^] (see also Q): 

B{t^bWg) ~ 0.3, (10) 
B{t bW-f) ~ 3.5 X 10^^ (11) 

The FCNC decays t ^ eg, t ^ and t ^ cZ occur at one loop, and are also GIM suppressed by a 
factor -S^ in the amplitude. Hence, the corresponding rates are very small M: 

(12) 
(13) 
(14) 

For a light Higgs boson, one can consider also the FCNC decay t cH (fig. |^) . A previous evaluation of 
its rates Q has now been corrected. For mn — 100 (160) GeV, one gets then |lOj: 

B(i ^ c77) ~ 0.9 X 10-13 (4 X 10-15). (15) 

We conclude this section, by presenting in table |l] a summary of the expected decay rates for the main 
top decay channels in the SM. 



B{t - 


^ eg) 


~ 5 X 10-" 


B{t - 


C7) 


~ 5 X IQ-i^ 


B{t - 


> cZ) 


~ 1.3 X 10-- 



3 Beyond the Standard Model (BSM) decays 

The fact that a measurement of the top width is not available and the branching ratio B{t ^ bW) is 
a model dependent quantity makes the present experimental constraints on the top BSM decays quite 
weak. Hence, the large value of rrit allows to consider the possibility of t decays into new massive states 
with branching fraction of order B{t ^ bW). Apart from the production of new final states with large 
branching fractions, we will see that new physics could also give rise to a considerable increase in the 
rates of many decay channels that in the SM framework are under the threshold of observability. 
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channel 


BsM 


bW 


1 


sW 


1.6 • 10"^ 


dW 


- lo-* 


bWg 


0.3 {Eg > 10 GeV) 


bW-y 


3.5 • 10"^ {E^ > 10 GeV) 


bWZ 


2 • lO-'* 


cW+W- 


- 10-^^ 


bW+H 


< 10-' 


19 


5 • 10-^^ 


11 


5 • 10-^^ 


qZ 


1.3 • 10"^^ 


cH 


< 10-^^ 



Table 1: Branching ratios for the main SM top decay channels. 



3.1 4 fermion family 

Extending the SM with a 4*'* fermion family can alter considerably a few t decay channels. First of 
all, when adding a 4*'' family to the CKM matrix the present constraints on the jVtgl elements are 
considerably relaxed. In particular, |Vts| and \Vtd\ can grow up to about 0.5 and 0.1, respectively |jl|. 
Correspondingly, assuming |Vtb| ^ 1 for the sake of normalization, one can have up to : 

Bi{t sW) - 0.25 and B^t dW) ~ 0.01, (16) 

to be confronted with the SM expectations in eq. (||). 

The presence of a 4*'' fermion family could also show up in the t direct decay into a heavy b' quark with 
a relatively small mass (mf,/ ~ 100 GeV) ||ll[. This decay would contribute to the t cWW channel. 
The corresponding rate would be: 

B{t W+b'{^ W-c)) - 10-3 (10-'^) at rub' = 100 (300) GeV, (17) 

to be confronted with the SM prediction in eq. (||). 

3.2 Two Higgs Doublet models (2HDM's) 

The possibility that the electroweak symmetry breaking involves more than one Higgs doublet is well 
motivated theoretically. In particular, three classes of two Higgs doublet models have been examined in 
connection with rare top decays, called model I, II and III. The first two are characterized by an ad hoc 
discrete symmetry which forbid tree- level FCNC [|l^ , that are strongly constrained in the lightest quarks 
sector. In model I and model II, the up-type quarks and down-type quarks couple to the same scalar 
doublet and to two different doublets, respectively (the Higgs sector of the MSSM is an example of model 
II). In model III ||l^, the above discrete symmetry is dropped and tree- level FCNC are allowed. In 
particular, a tree-level coupling tcH is predicted with a coupling constant < ^"^^J^" (where v is the 
Higgs vacuum expectation value). 

Since enlarging the Higgs sector automatically implies the presence of charged Higgs bosons in the spec- 
trum, one major prediction of these new frameworks is the decay t — > bH^ , with possibly competitive 
rates with B{t bW) for to^+ <, 170 GeV. In the MSSM, one has B{t hH+) - 1, both at smaU and 
large values of tan/3. If mu < mt — mb, one expects — > t+i^ (favored for large tan f3) and/or i7+ — > cs 
(favored for small tan/3) to be the dominant decays. Hence, for tan/3 > 1 and — > r+i/ dominant, one 
can look for the channel t — > bH^ by studying a possible excess in the t lepton signature from the t pair 
production On the other hand, if tan/3 < 2 and mn > 130 GeV, the large mass (or couphng) of the 
t-quark causes B{H~^ —^ t*b — > W^bb) to exceed B(i?+ — > cs) (see |l^ for details). As a consequence. 
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new interesting signatures at LHC like leptons plus multi-jet channels with four 6-tags, coming from the 
gluon-gluon fusion process gg tbH^ , followed by the H~ — » ib decay, have been studied These 
processes could provide a viable signature over a limited but interesting range of the parameter space. 
One should recall however that both B{t ^ and B{H^ W~^bb) are very sensitive to higher-order 

corrections, which are highly model dependent |l9tl . 

In model III, the tree-level FCNC decay t ch can occur with branching ratios up to lO^'^ |Q. In 
, the rate for the channel i — > eft. ^ cWW{cZZ, C77) has been studied. Accordingly, B{t cWW) 
can be enhanced by several orders of magnitude with respect to its SM value. In particular, for an 
on-shell decay with 2Mw ^ mh ^ mt, one can have up to B{t cWW) ^ 10^* from this source. The 
same process was considered in a wider range of models, where the decay t — > cWW can occur not only 
through a scalar exchange but also through a fermion or vector exchange In this framework, the 

fermion exchange too could lead to detectable rates for t cWW, as in eg. (|l7|). 

In 2IIDM's, the prediction for the FCNC decays t ^ eg, t ^ cj and t ^ cZ can also be altered. 
While in models I and II the corresponding branching fractions can not anyway approach the detectability 
threshold ||h in model III values up to B{t eg) ~ 10"^, B{t -> 07) ~ 10"'^ and B{t cZ) ~ 10"^ are 
predicted [ pi| . 

By further extending the 2HDM's sector and including Higgs triplets, one can give rise to a vertex 
HWZ at tree level in a consistent way ||2^. Accordingly, the t — > bWZ decay can be mediated by a 
charged Higgs (coupled with mt) that can enhance the corresponding branching fraction up to B{t — > 
bWZ) ~ 10"^. Large enhancements can also be expected in similar models for the channels t — *■ sWZ and 
t dWZ. 



3.3 Minimal Supersymmetric Standard Model (MSSM) 

Supersymmetry could affect the t decays in different ways [here, we assume the MSSM framework ||23|| , 
with (or without, when specified) R parity conservation]. 

First of all, two-body decays into squarks and gauginos, such as t ^ tig, t ^iX±_> t ^ ^iXi; could 
have branching ratios of order B{t ^ bW), if allowed by the phase space (sec, i.e., |2J] for references). 
QCD corrections to the channel t —f iig have been computed in ]25[ | and found to increase its width up 
to values even larger than r(t — > bW). Three-body t decays in supersymmetric particles were surveyed 
in||]. 

The presence of light top and bottom squarks, charginos and neutralinos in the MSSM spectrum could 
also give rise to a CP asymmetry of the order 10^'^ in the partial widths for the decays t — > bW^ and 
i bW- |§. 

Explicit i?-parity violating interactions p7| ] could provide new flavor-changing t decays, both at tree- 
level (as in the channels t fb and t rbxi |^^) and at one loop (as int^ci) |2^), with observable 
rates. For instance, B{t cv) ~ 10~^ — 10"'^ in particularly favorable cases. 

Another sector where supersymmetric particles could produce crucial changes concerns the one-loop 
FCNC decays t ^ eg, t — > e7, t ~* cZ and t cH , which in the SM are unobservally small. In the 
MSSM with universal soft breaking the situation is not much affected, while, by relaxing the universality 
with a large flavor mixing between the 2"'^ and 3'''' family only, one can reach values such as IT] : 

BMSSM{t-^cg) ^ 10-6 (18) 
BMSSM{t^c^) - 10-^ (19) 
BMSSAiit^cZ) ^ 10-^ (20) 

which anyhow are still not observable. The introduction of _B-violating couplings in broken i?-parity 
models could on the other hand give large enhancements , and make some of these channels observable. 
The corresponding upper limits on branching ratios get then: 



.eg) ^ 10-^ (21) 
Bffit^c-i) - 10-5 (22) 
-cZ) - lO-''. (23) 
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A particularly promising channel is the FCNC decay t ^ ch in the framework of MSSM, where 
h = h^, H'^ , is any of the supersymmetric neutral Higgs bosons |Q. By including the leading MSSM 
contributions to these decays (including gluino-mediated FCNC couplings), one could approach the de- 
tectability threshold, especially in the case of the light CP-even Higgs boson, for which one can get up 
to: 

BMSSM{t ^ ch") ^ (24) 



4 LHC potential for top rare decays 

An extensive analysis of the LHC potential for detecting top rare decays has been performed by both 
the LHC experiments ATLAS and CMS [Q. In the framework of the SM, the top rare decays (that is 
any channel different from t — > qW) turn out to be definitely below the threshold for an experimental 
analysis at LHC. On the other hand, LHC experiments will be able to probe quite a few predictions of 
possible extensions of the SM. 

An extended Higgs sector will be looked for through the tree-level decay t hH^ . ATLAS estimates 
its sensitivity to this channel in the MSSM, through an excess in the tau lepton signal, to be around 
B(t — > H'^h) ~ 3% (that is almost 4 times better than what expected from Run 2 at the Tevatron). This 
would allow to probe all values of mu± below mj — 20 GeV over most of the tan (3 range. For low tan /3, 
the complementary decay mode cs has been considered. In the mass range 110 < < 130 GeV, 

the peak can be reconstructed and separated from the dominant W — > jj background. 
For CMS, using the tau excess signature, the expected Su discovery range for 10 fb^^ in the MSSM 
{niA, tan (3) parameter space is niA < 110 GeV, for all tan/3 values, and somewhat extended {ttia ^ 
140), for tan^ < 2. 

Other interesting signatures like hW* , AW* and bt* bbW are very promising in 

particular parameter ranges, but have not yet been thoroughly investigated. 

ATLAS has studied its sensitivity to the radiative decay t WbZ . This has been found to be at 
most of the order 10~'*, that is insufficient for the study of a SM signal 10^®), but possibly useful for 
exploring the predictions of some extended Higgs-sector model, for which B(< ^ WqZ) ^ lO^'^. On the 
other hand, the radiative Higgs decay t WbH seems out of the reach of LHC in any realistic model. 

The LHC reach for the FCNC decays t qZ, t ^ q^ and t — > qg has also been thoroughly 
investigated. Apart from the t qg , which is completely overwhelmed by the hadronic background, 
both ATLAS and CMS have a sensitivity of about 2 x 10~^ to the t ^ qZ channel, while the CMS 
reach for the t ^ qj channel is about 3.4 x 10~^, that is slightly better than the ATLAS sensitivity 
(1.0 X lO^'*), assuming an integrated luminosity of 100 fb^^. These thresholds are too high to test the 
predictions of the models reviewed here. On the other hand, they could be largely sufficient to detect 
some manifestation of possible FCNC anomalous couplings in the top sector Q. 
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